Image processor

ABSTRACT

There is provided an image processor and a method thereof for high-speed compensation for taken-image blurs produced by camera shakes or the like. In the first instance, a motion-detecting area is selected for each of two images taken by an image sensor. When projective data is calculated by means of computing in a predetermined direction pixels of the motion-detecting areas, the motion vector between the two images can be acquired based on the projective data. The image correlativity between the two images is then calculated in the direction that the motion vector designates; and the amount of pixel displacement between the two images is calculated based on the correlativity values acquired by the calculation. Moreover, the area that has been produced by displacing an image output area in a camera-shake compensation area designated in the second frame, by the pixel-displacement amount calculated by a displacement calculator is cut away from the camera-shake compensation area, and is outputted as an image for the image output area of the second frame.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to image processors and processing methodsthereof for treating blurs in taken images produced by camera shakes orthe like, and particularly to reduction in image-processing timerequired to compensate the blurs.

2. Description of the Related Art

A great number of digital and movie cameras equipped with an imagesensor, such as a CCD (Charge Coupled Device) sensor and a CMOS(Complementary Metal Oxide Semiconductor) sensor, have recently beenmarketed, and even beginners have been able to readily take stillpictures or moving pictures; however, when a general user takes apicture, blurs are inevitably produced in the picture unless a camera isfixed with a tripod or the like; in consequent, various methods havebeen devised in order to compensate the blurs.

The methods are roughly categorized into two groups. One of the methodsis to mechanically control the orientations of lenses and prisms bydetecting with an acceleration sensor or the like movement of a cameraitself, and the other is to compensate image blurs by applying imageprocessing to the digital data of a taken picture and calculatingblurring amounts.

Despite its high accuracy of compensating blurs, the former requires anacceleration sensor and parts for lens control; therefore, there areproblems in terms of downsizing and cost reduction of the device. Incontrast, despite its disadvantage of deterioration in resolution, thelatter can, merely by means of signal processing, provide thecompensation for camera shakes, and can be implemented with low cost;therefore, the latter method has been widely utilized.

Japanese Laid-Open Patent Publication 1993-75913 discloses amotion-vector detecting and camera-shake compensating circuits that areexamples of conventional arts with regard to the camera-shakecompensation utilizing the image processing. In the foregoing JapaneseLaid-Open Patent Publication, an input image is divided into a pluralityof areas, and motion vectors in selected areas among the plurality ofregions are calculated. If the number of the selected areas is increasedin order to raise the accuracy of the motion vectors, the calculatingamount of the motion vectors increases; therefore, a circuit fordetermining the reliability of correlativity calculation is provided foreach area, and the motion-vector calculation is limited to the areasthat are determined by the determining circuit to be reliable; however,when all the areas are reliable, the motion-vector calculation is alsocarried out in all the areas; thus, processing speed has been an issueto be solved.

SUMMARY OF THE INVENTION

In the present invention, a motion-detecting area, an image output area,and a camera-shake compensation area are set in a two-dimensional imagetaken by an image sensor. In the first instance, the calculation ofprojective data, for two images taken by the image sensor, throughcomputing pixel values in the motion-detecting areas in a predetermineddirection provides, based on the projective data, a motion vectorbetween the two images. The image correlativity between an image in theimage output area of the first frame and an image in the camera-shakecompensation area of the second frame is then calculated in thedirection that the motion vector designates; and the amount of pixeldisplacement between the two images is calculated based on thecorrelativity values acquired by the calculation. Moreover, an area thatis displaced by the amount of pixel displacement is cut away from thecamera-shake compensation area of the second frame, and is outputted asan image in the image output area of the second frame. According to thepresent invention, an accurate and high-speed camera-shake compensationfunction can be implemented.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating the configuration of an imageprocessor according to Embodiment 1 of the present invention;

FIG. 2 is a flowchart illustrating the processing procedure according toEmbodiment 1 of the present invention;

FIG. 3 is a view for explaining the operation of a motion calculator;

FIG. 4 is a view for explaining the operation of a displacementcalculator;

FIG. 5 is a view for explaining the relationship between a basic imagearea and rectangular areas;

FIG. 6 is a block diagram illustrating the configuration of an imageprocessor according to Embodiment 2 of the present invention;

FIG. 7 is a flowchart illustrating the processing procedure according toEmbodiment 2 of the present invention; and

FIG. 8 is a view for explaining the operation of aconversion/compensation unit.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment 1

Embodiment 1 will be discussed below referring to the accompanyingdrawings. FIG. 1 is a block diagram illustrating the configuration of animage processor according to Embodiment 1 of the present invention, andillustrates four elements—an image input unit 1, a motion calculator 2,a displacement calculator 3, and an image output unit 4. FIG. 2 is aflowchart illustrating the procedure of image compensation.

In the first instance, the image input unit 1 acquires two-dimensionalframe images by means of an image sensor such as a CCD sensor or a CMOSsensor (ST101). The motion calculator 2 calculates motion vectors fortwo consecutive frame images, which have been acquired by the imageinput unit 1, by utilizing designated motion-detecting areas (ST102).The displacement calculator 3 computes the amount of pixel displacementbetween the two frame images, by calculating the image correlativitybetween the two frame images, in a limited direction that a motionvector calculated by the motion calculator 2 designates (ST104). Theimage output unit 4 cuts away from a camera-shake compensation area ofthe second frame an area that has been produced by displacing an imageoutput area, by the amount of previously calculated pixel displacement,and outputs the area as a compensated image for the image output area ofthe second frame (ST105).

The operation of the motion calculator 2 will be more particularlydiscussed referring to FIGS. 3 (a) through 3 (d). In FIG. 3 (a), thefour rectangular areas illustrated with diagonal lines denotemotion-detecting areas 6. In each of motion-detecting areas 6,projective data is acquired by adding up pixel values in the vertical orhorizontal direction in which pixels form rows. In this case, thediscussion is being made by exemplifying the case where the number ofmotion-detecting areas 6 is four; however, five or more, or one to threemotion-detecting areas 6 may be provided.

FIG. 3 (b) shows vertical projective data 7 that is acquired by addingup pixel values in the vertical direction, and horizontal projectivedata 8 that is acquired by adding up pixel values in the horizontaldirection. In this situation, ten pixels each are included in thevertical and horizontal directions. By utilizing the vertical projectivedata 7 and the horizontal projective data 8, and by calculating theimage correlativity between two frame images, image motion vectorsincluded in the motion-detecting areas 6 can be acquired.

The method for acquiring the horizontal components of the motion vectorswill be particularly explained below, by utilizing vertical projectivedata of two frames. In the first instance, vertical projective data 7 ofthe first frame is illustrated in FIG. 3 (c); and vertical projectivedata 11 of the second frame, in FIG. 3 (d). In FIG. 3 (c), thefirst-frame projective data that is subject to correlativity calculationis included over a range w1; and the first-frame projective data that isnot subject to correlativity calculation, over ranges w2. Moreover, inFIG. 3 (d), the leftmost projective data among the projective data ofthe second frame, which is subject to correlativity calculation, isincluded over a range 9; and the rightmost projective data, over a range10. The ranges 9 and 10 each have a width of w1.

The image immediately after the first-frame image is generally chosen asthe second-frame image; however, the images that are a predeterminednumber of frames apart from each other may be chosen. The correlativitycalculation between projective data (w1) of the first frame andprojective data of the second frame is carried out, displacing eitherone of the projective data pixel by pixel over the ranges 9 and 10.

In particular, with regard to the first-frame projective data over therange w1 and the second-frame projective data, the absolute values ofdifference between corresponding projective data are added up, and theresultant sum is employed as a correlativity value. When correlativitycalculation is carried out with projective data being displaced pixel bypixel, the location where the correlativity value of the verticalprojective data, which is calculated for each place, takes a minimalvalue corresponds to the horizontal component of a motion vector. Forexample, with respect to the projective data of the first frame, if aminimal value is produced at a position within the range 9 in theprojective data of the second frame, the horizontal component of amotion vector to be acquired is −w2; and if within the range 10, +w2.The same calculation applied to the horizontal projective data 8provides the vertical component of a motion vector.

In the foregoing manner, a single motion vector is acquired from each ofthe four motion-detecting areas 6. A motion vector that represents thedirection in which a camera equipped with an image sensor has moved iscalculated with the four motion vectors acquired in this manner. Forthat purpose, the average of the four motion vectors, or the average ofmotion vectors acquired by other methods, e.g., by comparing thedirection angles of the four motion vectors and eliminating, as noise, nmotion vectors (n is less than 3) that are extremely different in angleand size, and by averaging out (4−n) other motion vectors, may beemployed as the representative motion vector.

The operation of the displacement calculator 3 will be discussedreferring to FIGS. 4 (a) through (e). FIG. 4 (a) shows a sensor pixelarea 16 that is a total area of an imaging device, and an image outputarea 12 that is subject to application of camera-shake compensation. Theimage output area 12 is displayed to a user. A camera-shake compensationarea 13 embraces the image output area 12 and a diagonal-line areaillustrated around the image output area 12; a rectangular area havingthe same size as that of the image output area 12 is cut away from thecamera-shake compensation area 13.

When the camera-shake compensation is applied, images of two frames areutilized and the correlativity between the two images is acquired. Inthis regard, however, an image in the image output area 12 is utilizedas the first-frame image; and an image cut away from the camera-shakecompensation area 13, as the second-frame image. In other words,extracting from the camera-shake compensation area 13 an image area thathas the same size as that of the image output area 12 and that has thegreatest correlativity amounts to obtaining a moving image with camerashakes being compensated, when the two images are viewed as time-seriesimages.

In order to calculate the correlativity between the respective images ofthe output area 12 and the camera-shake compensation area 13, the amountof calculation is reduced by selectively thinning out pixels. FIGS. 4(b) through (e) shows the concrete examples of the selection methods;FIG. 4 (b) is the method in which pixels are thinned out every severalpixels and are uniformly selected over the image output area 12; FIG. 4(c) is the method in which the thinning-out space in FIG. 4 (b) isextended; FIG. 4 (d) is the method in which pixels to be selected arethinned out and the selection areas are limited to four corners; andFIG. 4 (e) is the method in which the selection areas are limited toareas other than the four corners.

In acquiring image correlativity, the amount of calculation is furtherreduced by thinning out and selecting the pixels that are subject toapplication of the camera-shake compensation, as described above, and bylimiting the direction in which image correlativity is calculated to asingle direction based on the motion vector calculated by the motioncalculator 2. In other words, as illustrated in FIG. 5, in selecting themost correlative area within the camera-shake area 13 of the secondframe, with respect to the basic image 12 of the first frame, imagecorrelativity is calculated with regard to rectangular areas 15 thatalign along the direction 14 of the motion vector calculated by themotion calculator 2 and that have the same size as that of the imageoutput area 12, and the most correlative area among the rectangularareas 15 is selected.

When, as a concrete way of calculating image correlativity, the absolutevalues of the differences between the values of corresponding pixels arecalculated, and the sum of the absolute values is acquired as acorrelativity value, the correlativity value takes its minimal value inthe area that has the greatest correlativity. Accordingly, bycalculating correlativity values along the direction of the motionvector 14, and by selecting among the rectangular areas an area thatprovides the minimal value, the most correlative area can be selected.

Because the image output unit 4 outputs, as an image for the imageoutput area 12 of the second frame, a rectangular area having thestrongest correlativity with the image that has been subject to thecalculation by the displacement calculator 3, and that is in the imageoutput area 12 of the first frame, a camera-shake-compensated image forthe image output area 12 in FIG. 3 (a) can be obtained.

The image processing method for taken images discussed above isimplemented by image-compensation programs, which are provided, beingrecorded in recording media that can be read by a computer or the like.

In addition, in the motion calculator 2, with regard to the acquisitionof projective data that the motion-vector calculation requires, the casehas been exemplified, wherein the sensor pixel area 16 is subject to thedata-acquisition; however, the projective data may be acquired from animage in a partial image area that is cut away from an input image takenby the image input unit 1. Moreover, the calculation of projective datamay be implemented by an ASIC (Application Specific IC), and the motionvectors may be calculated by utilizing the projective data acquired inthis manner.

Still moreover, in the motion calculator 2, the motion vectors have beenacquired through correlativity calculation without increasingresolution, i.e., the number, of projective data for motion-vectorcalculation; however, the same motion-vector calculation with theresolution of the projective data being raised by means of interpolationimproves the accuracy of calculated motion vectors, and the comparisonrange of image-correlativity calculation in the displacement calculator3 can be reduced.

As discussed above, according to Embodiment 1, the motion calculator 2calculates motion vectors for an image taken by the image input unit 1;the displacement calculator 3 calculates the amount of pixeldisplacement between two frames, by limiting the image-correlativitycalculation in the directions of motion vectors and thereby reducing theamount of its calculation; and the image output unit 4 creates an outputimage, by cutting a partial area with the amount of pixel displacementbeing cancelled out; therefore, an image with pixel displacementproduced by camera shakes being compensated can be obtained at highspeed.

Embodiment 2

Embodiment 2 will be discussed below, referring to the drawings. FIG. 6is a block diagram illustrating the configuration of an image processoraccording to Embodiment 2 of the present invention, and the presentimage processor is constituted from the five units, including aconversion/compensation unit 5 in addition to the constituent elementsin Embodiment 1. Moreover, FIG. 7 is a flowchart illustrating theprocedure of image compensation.

The operation of the image input unit 1, the motion calculator 2, thedisplacement calculator 3, and the image output unit 4 is the same asthat in Embodiment 1. The conversion/compensation unit 5 detectspivoting and zooming components in a plurality of motion vectorscalculated by the motion calculator 2, and applies the pivoting andzooming conversion to an input image, by utilizing the motionalcomponents.

The operation of the conversion/compensation unit 5 will particularly bediscussed below. FIG. 8 is a view for explaining a method for detecting,from four motion vectors calculated by the motion calculator 2, camerapivoting on the shot axis and camera expanding/contracting (zooming)movement associated with advancing and retreating along the shot axis.The camera shake discussed in Embodiment 1 is translational movement inparallel with the imaging area of a camera; however, it is an object ofEmbodiment 2 to detect, as the movement of a camera, pivoting andzooming components other than translational components. The case will bediscussed below, where the movement of a camera itself consists merelyof pivoting and zooming components.

FIG. 8 (a) illustrates patterns of four motion vectors in the case wherethe movement of a camera consists merely of pivoting components; andFIG. 8 (b) illustrates patterns of four motion vectors in the case wherethe movement of a camera consists merely of zooming components. In caseswhere only pivoting components or zooming components independentlyappear without being mixed with each other, pivoting or zoomingcomponents are quantitatively calculated utilizing the strength of eachmotion vector.

As in FIG. 8 (c), in cases where pivoting or zooming components aremixed up, each motion vector is separated into pivoting and zoomingcomponents, and then each motional component is calculated. FIG. 8 (d)is a view illustrating a method for calculating pivoting and zoomingcomponents for the motion vectors, which consist of pivoting and zoomingcomponents, as illustrated in FIG. 8 (c); the solid lines denoteauxiliary lines for calculating pivoting components, and dotted lines,for calculating zooming components. In other words, when the four motionvectors are projected on the respective auxiliary lines that areperpendicular to each other, and when the vectors projected are acquiredas the pivoting and zooming components of the respective motion vectors,the calculation can be carried out, in the same way as in the case whererespective components independently appear.

In order to cancel out each motional component of a camera, by utilizingpivoting and zooming components that are quantitatively calculatedthrough the foregoing procedure, conversion in the reverse direction tothe pivoting and zooming components detected is applied to the acquireddata for an image in the second frame. In addition, with regard to eachof the components, the correlativity between the data for thesecond-frame image and the data for the first-frame image is calculatedby converting the data for the second frame image in constantincrements, e.g., by single degrees for pivoting components and bymultiples of 0.1 for zooming components. The selection of thesecond-frame image that has the highest correlativity amounts toreproducing an image whose pivoting and zooming components have beencancelled out (ST103 a).

Because actual motion vectors include pivoting and zooming components,the subtraction of the pivoting and zooming components acquired by theforegoing procedure from the motion vectors leaves only translationalcomponents that are in parallel with the imaging area of a camera. Themotion vectors acquired in this manner are hereinafter referred to ascompensated motion vectors. In the same manner as in Embodiment 1, arepresentative motion vector, which indicates the direction in which acamera equipped with an image sensor has moved, is calculated, byutilizing the compensated motion vectors (ST103 b).

Next, the displacement calculator 3, by utilizing the representativemotion vectors, selects the most correlative rectangular area within thecamera-shake area 13 of the second frame, with respect to the locationof the basic image area 12 of the first frame, as illustrated in FIG. 5.In addition, by utilizing the reproduced second-frame image whosepivoting and zooming components have been cancelled out and thefirst-frame image in the basic image area, motion vectors can beacquired again.

Because the image output unit 4 outputs, as an image for the imageoutput area 12 of the second frame, a rectangular area that has beencalculated to have the strongest correlativity with the image in theimage output area 12 of the first frame, an image compensated for camerashakes with regard to the image output area 12 can be obtained.

The image processing method for taken images discussed above isimplemented by image-compensation programs, which are provided beingrecorded in recording media that can be read by computers or the like.

The present invention relates to the technology for compensating blursin taken images by means of image processing in order to prevent thedeterioration of two-dimensional images, which is caused by camerashakes produced during taking pictures by the camera, and the presentinvention can be applied to digital cameras and movie cameras that alloweven beginners to readily enjoy still or moving images whose blurs arenot conspicuous.

Because this invention may be embodied in several forms withoutdeparting from the spirit of the essential characteristics thereof, thepresent embodiments are therefore illustrative and not restrictive,since the scope of the invention is defined by the appended claimsrather than by the description preceding them, and all changes that fallwithin the metes and bounds of the claims, or the equivalence of suchmetes and bounds, are therefore intended to be embraced by the claims.

1. An image processor comprising: an image input unit configured toreceive two-dimensional images; a motion calculator configured to selecta plurality of motion detecting areas for each of two images received bythe image input unit, and configured to calculate motion vectors betweenthe two images, with regard to each of the plurality of motion detectingareas, based on projective data that is acquired by computing, in apredetermined direction, pixel values in the motion detecting areas; aconversion/compensation unit configured to calculate pivoting andzooming components by means of the plurality of motion vectorscalculated by the motion calculator, and configured to apply pivotingand zooming conversion to the second image, based on the pivoting andzooming components, and configured to acquire a compensated motionvector by subtracting the pivoting and zooming components from theplurality of motion vectors; a displacement calculator configured tocalculate the image correlativity between the two images, in a directionthat the compensated motion vector designates, and configured tocalculate the amount of pixel displacement between the two images, basedon the correlativity calculations; and an image output unit configuredto cut away an area from a camera-shake compensation area designated ina frame, the area being produced by displacing an image output area inthe camera-shake compensation area, by the pixel-displacement amountcalculated by the displacement calculator, and configured to output thearea as an image for the image output area of the frame.
 2. An imageprocessing method implemented on an image capture device, comprising:receiving two-dimensional images; selecting a plurality of motiondetecting areas for each of two received images, and calculating motionvectors between the two images, with regard to each of the plurality ofmotion detecting areas, based on projective data that is acquired bycomputing, in a predetermined direction, pixel values in the motiondetecting areas; calculating pivoting and zooming components by means ofthe plurality of calculated motion vectors, and applying pivoting andzooming conversion to the second image based on the pivoting and zoomingcomponents; calculating a compensated motion vector by subtracting thepivoting and zooming components from the plurality of motion vectors;calculating the image correlativity between the two images, in adirection that the compensated motion vector designates, and calculatingthe amount of pixel displacement between the two images, based on thecorrelativity calculations; and cutting away an area from a camera-shakecompensation area designated in a frame, the area being produced bydisplacing an image output area in the camera-shake compensation area,by the pixel-displacement amount, and outputting to a display the areaas an image for the image output area of the frame.
 3. A recording mediaencoded with an image compensation program configured to cause aninformation processing apparatus to execute a method, the methodcomprising: selecting a plurality of motion detecting areas for each oftwo received images, and calculating motion vectors between the twoimages, with regard to each of the plurality of motion detecting areas,based on projective data that is acquired by computing, in apredetermined direction, pixel values in the motion detecting areas;calculating pivoting and zooming components by means of the plurality ofcalculated motion vectors, and applying pivoting and zooming conversionto the second image based on the pivoting and zooming components;calculating a compensated motion vector by subtracting the pivoting andzooming components from the plurality of motion vectors; calculating theimage correlativity between the two images, in a direction that thecompensated motion vector designates, and calculating the amount ofpixel displacement between the two images, based on the correlativitycalculations; and cutting away an area from a camera-shake compensationarea designated in a frame, the area being produced by displacing animage output area in the camera-shake compensation area, by thecalculated pixel-displacement amount, and outputting the area as animage for the image output area of the frame.